{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 如何使用千帆 Python SDK 搭配预置大模型服务进行批量推理\n",
    "\n",
    "在 0.2.4 中，千帆 Python SDK 增加了对批量推理的支持，使用该功能需要视情况开通千帆的模型服务，确保您的账号可以调用您想进行批量推理的服务。\n",
    "\n",
    "# 准备工作\n",
    "\n",
    "在开始之前，请确保你的千帆 Python SDK 已经升级到了 0.2.4 及以上版本。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "pip install -U \"qianfan>=0.2.4\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "并且在环境变量中设置好 Access Key 与 Secret Key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ['QIANFAN_ACCESS_KEY'] = 'your_access_key'\n",
    "os.environ['QIANFAN_SECRET_KEY'] = 'your_secret_key'\n",
    "os.environ[\"QIANFAN_QPS_LIMIT\"] = \"4\"\n",
    "os.environ['QIANFAN_CONSOLE_API_RETRY_COUNT'] = \"10\"\n",
    "os.environ['QIANFAN_CONSOLE_API_RETRY_TIMEOUT'] = \"60\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 正文\n",
    "\n",
    "为了开始批量推理，我们首先需要获取到用于做批量推理输入的数据集文件，并且指定用做推理输入的列名"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'prompt': '地球的自转周期是多久？', 'response': '大约24小时'}\n"
     ]
    }
   ],
   "source": [
    "from qianfan.dataset import Dataset\n",
    "\n",
    "dataset_file_path = \"data_file/qa_pair.csv\"\n",
    "dataset_input_column_list = [\"prompt\"]\n",
    "\n",
    "ds = Dataset.load(data_file=dataset_file_path, input_columns=dataset_input_column_list)\n",
    "\n",
    "# 如果用户需要使用对话类数据集进行批量推理，还需要指定应用列\n",
    "# reference_column = \"column3\"\n",
    "# ds = Dataset.load(data_file=dataset_file_path, input_columns=dataset_input_column_list, reference_column=reference_column)\n",
    "\n",
    "# 预览数据格式\n",
    "print(ds.list(0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在导入之后，用户可以根据自己的需求，传入不同的参数来使用不同的方式进行推理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 用户可以设置 service_model 为自己想要的模型名，来直接对数据进行批量推理，以 EB 4 为例\n",
    "result = ds.test_using_llm(service_model=\"ERNIE-Bot-4\")\n",
    "\n",
    "# 用户还可以设置 service_endpoint 来使用预置或自己的服务。\n",
    "result = ds.test_using_llm(service_endpoint=\"/chat/completions_pro\")\n",
    "\n",
    "# 如果是使用对话类数据集进行批量推理，需要设置 is_chat_service=True\n",
    "result = ds.test_using_llm(service_model=\"ERNIE-Bot-4\", is_chat_service=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "拿到的 `result` 对象也是一个 `Dataset` 对象，可以继续使用千帆 Python SDK 进行后续处理，或者直接保存到本地。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'prompt': '地球的自转周期是多久？', 'input_prompt': '地球的自转周期是多久？', 'llm_output': '地球的自转周期是**23小时56分**。'}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(result.list(0))\n",
    "      \n",
    "dataset_save_file_path = \"output_file.csv\"\n",
    "\n",
    "result.save(data_file=dataset_save_file_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 进阶能力\n",
    "\n",
    "在调用 `test_using_llm` 时，用户还可以传入一些额外参数来支持额外的功能，比如设置 Prompt 模板，设置人设字段，或者传入大模型调用时的超参数\n",
    "\n",
    "当进行的是非对话类推理时，用户可以传入 `prompt_template` 参数来传递一个 Prompt 模板。`prompt_template` 是一个千帆 Python SDK 的 `Prompt` 对象，用户可以通过设置 `Prompt` 对象的 `template` 成员来自定义被用于推理的模板，模板渲染出来的内容将会被作为最终输入提交给大模型。以示例数据集为例，我们可以这么指定一个模板："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from qianfan.common import Prompt\n",
    "\n",
    "prompt = Prompt(template=\"请你就以下问题进行回答: {prompt}\")\n",
    "\n",
    "# 传递给函数\n",
    "result = ds.test_using_llm(service_model=\"ERNIE-Bot-4\", prompt_template=prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "除此之外，用户还可以传入 `system_prompt` 参数来指定对话中大模型需要遵守的人设"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = ds.test_using_llm(service_model=\"ERNIE-Bot-4\", system_prompt=\"人设 prompt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用户在进行批量推理时，还可以直接在 test_using_llm 中传入模型支持的超参数，例如我们可以这么设置模型的 `temperature` :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = ds.test_using_llm(service_model=\"ERNIE-Bot-4\", system_prompt=\"人设 prompt\", temperature=0.1)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "bce-qianfan-sdk",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
